Skip to content
This repository was archived by the owner on Sep 2, 2026. It is now read-only.

Refactor the code for the netlify functions #296

Merged
theory-in-progress merged 7 commits into from
Aug 22, 2023
Merged

Refactor the code for the netlify functions #296

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
7 commits
Select commit Hold shift + click to select a range
c4e612a
Refactor the code for the netlify functions
theory-in-progress Aug 20, 2023
c68f16f
Removed hash and created Uid from the content of zip file
theory-in-progress Aug 21, 2023
3336f27
Using the store.config for generating the unique id
theory-in-progress Aug 22, 2023
cb0b9ad
Pass the config in the body to the function
theory-in-progress Aug 22, 2023
0319b43
Convert to string
theory-in-progress Aug 22, 2023
012997d
use JSON.stringify()
theory-in-progress Aug 22, 2023
311a1de
Merge branch 'main' into refactor-zipRes
vfdev-5 Aug 22, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
31 changes: 2 additions & 29 deletions functions/code.js
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,39 +1,12 @@
// Below imports are defined in
// `external_node_modules` of [functions] in netlify.toml
// They are required for this function to run

import { v4 as uuidv4 } from 'uuid'
import JSZip from 'jszip'
import { pushToGitHub } from './utils'

const nbUid = uuidv4()
const repoOwner = process.env.VUE_APP_GH_USER
const repo = process.env.VUE_APP_GH_REPO
import { getZip_Uid } from './utils'

// This function is the one Netlify function runs on
// https://docs.netlify.com/functions/build-with-javascript/#synchronous-function-format
exports.handler = async function (event, _) {
// event is a JSON object
const data = JSON.parse(event.body)
const zip = new JSZip()
const code = data.code
const template = `ignite-${data.template}`
const { zipRes, nbUid } = await getZip_Uid(data)

// As usual from Download component,
// we will zip the files and
// generate a base64 format for pushing to GitHub
// with Octokit.
for (const filename in code) {
zip.file(filename, code[filename])
}
const content = await zip.generateAsync({ type: 'base64' })
const zipRes = await pushToGitHub(
content,
`${template}.zip`,
nbUid,
repoOwner,
repo
)
return {
statusCode: 200,
body: zipRes
Expand Down
34 changes: 3 additions & 31 deletions functions/colab.js
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,10 +1,4 @@
// Below imports are defined in
// `external_node_modules` of [functions] in netlify.toml
// They are required for this function to run

import { v5 as uuidv5 } from 'uuid'
import JSZip from 'jszip'
import { pushToGitHub } from './utils'
import { pushToGitHub, getZip_Uid } from './utils'

const repoOwner = process.env.VUE_APP_GH_USER
const repo = process.env.VUE_APP_GH_REPO
Expand All @@ -14,29 +8,9 @@ const repo = process.env.VUE_APP_GH_REPO
exports.handler = async function (event, _) {
// event is a JSON object
const data = JSON.parse(event.body)
const zip = new JSZip()
const code = data.code
let hash = ''
const template = `ignite-${data.template}`
const nbName = `${template}.ipynb`

// As usual from Download component,
// we will zip the files and
// generate a base64 format for pushing to GitHub
// with Octokit.
for (const filename in code) {
hash += code[filename]
zip.file(filename, code[filename])
}
const nbUid = uuidv5(hash, uuidv5.URL)
const content = await zip.generateAsync({ type: 'base64' })
const zipRes = await pushToGitHub(
content,
`${template}.zip`,
nbUid,
repoOwner,
repo
)
const { zipRes, nbUid } = await getZip_Uid(data)

const title = template
.replace('ignite-', '')
Expand Down Expand Up @@ -101,9 +75,7 @@ exports.handler = async function (event, _) {
await pushToGitHub(
Buffer.from(JSON.stringify(nb)).toString('base64'),
nbName,
nbUid,
repoOwner,
repo
nbUid
)

const colabLink = `https://colab.research.google.com/github/${repoOwner}/${repo}/blob/main/nbs/${nbUid}/${nbName}`
Expand Down
34 changes: 33 additions & 1 deletion functions/utils.js
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,11 @@
// They are required for this function to run

import { Octokit } from '@octokit/core'
import JSZip from 'jszip'
import { v5 as uuidv5 } from 'uuid'

const repoOwner = process.env.VUE_APP_GH_USER
const repo = process.env.VUE_APP_GH_REPO

/**
* Create a file on GitHub with Octokit.
Expand All @@ -13,7 +18,7 @@ import { Octokit } from '@octokit/core'
* @param {string} repo
Comment thread
guptaaryan16 marked this conversation as resolved.
Outdated
* @returns download_url
*/
export async function pushToGitHub(content, filename, nbUid, repoOwner, repo) {
export async function pushToGitHub(content, filename, nbUid) {
const octokit = new Octokit({
auth: process.env.VUE_APP_GH_TOKEN
})
Expand All @@ -33,3 +38,30 @@ export async function pushToGitHub(content, filename, nbUid, repoOwner, repo) {
console.error(e)
}
}

// This function is the one Netlify function runs on
// https://docs.netlify.com/functions/build-with-javascript/#synchronous-function-format
export async function getZip_Uid(data) {
const zip = new JSZip()
const code = data.code
let hash = ''
const template = `ignite-${data.template}`

// As usual from Download component,
// we will zip the files and
// generate a base64 format for pushing to GitHub
// with Octokit.
// we generate a hash for unique code identification and
// zip generation
for (const filename in code) {
hash += code[filename]
zip.file(filename, code[filename])
}
const nbUid = uuidv5(hash, uuidv5.URL)
const content = await zip.generateAsync({ type: 'base64' })

@vfdev-5 vfdev-5 Aug 20, 2023

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We can also produce the hash from the content instead of concatenated files with code. To see if this makes more interesting in terms of perfs. I suspect that content length could be smaller than the length of hash.
By the way, the name of variable hash is wrong as we do not create any hash with hash += code[filename].

@theory-in-progress theory-in-progress Aug 21, 2023 •
edited
Loading

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It is indeed smaller than concatenating the whole code. But I guess one thing I didn't understand while testing was that the content is different for the same template every time, without any modifications... I thought same code with the same files would generate the same zip.

So it's also generating a new Uid for each open in colab button press, which it should not for no code changes... This leads me to believe that zip being generated is slightly different every time. This creates a new uid each time, Which we wouldn't want. Ideally, we would like to keep the uid same for the template without any changes...

@guptaaryan16 since you've worked on this, do you have any thoughts?

@guptaaryan16 guptaaryan16 Aug 21, 2023

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@theory-in-progress @vfdev-5 Actually when we click on 'Open in Colab' or 'Download Zip', it makes an async request to the netlify functions to get the uuid for the code and the code link. I think we should create a store.js: uuid variable which gets updated to NULL whenever we make any changes in the store.config and these requests to netlify functions should be made only when store.uuid != NULL. Probably needs a fresh PR on this

@theory-in-progress theory-in-progress Aug 21, 2023

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

But these netlify functions are triggered only when we click on 'Open in Colab' or 'Download Zip'. And each time they are being for the changed code, we would like it to contain a new uuid. Also the gencode function is on watch for dynamic updates to the config so I'm unsure on how to set a global variable shared between the three files which would indicate that the config has been updated.

@guptaaryan16 guptaaryan16 Aug 21, 2023

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Probably we can update the functions to take uuid parameter from store.js in NavCode and NavDownload such that the call to save template and make uuid request only happens when uuid != NULL otherwise it will just use the uuid and create a link using that itself. Also if config changes we can update the uuid = NULL

const zipRes = await pushToGitHub(content, `${template}.zip`, nbUid)
return {
zipRes: zipRes,
nbUid: nbUid
}
}