Just blame the cache

This is the story about how Netlify caching bit me in the ass. I was trying to add firestore, a google database, to Find Communities Today because I wanted to add voting and comments to the communities to give users a better view/opinion of a community.

The database document would look pretty simple. It would only have the name of the community, the number of likes, the number of dislikes and finally an array with all the comments.

I spent a few hours reading the firebase/firestore documentation and how to include it into gatsby. Since there was a plugin for it already, it was easy to make firestore work with Find Communities Today. After reading the firestore documentation I was able to write a few lines of code quickly to create the document into the database when Node creates the pages for each community.

I've run the command gatsby develop and surprise! Everything worked just fine!

The firestore had now 120 documents - one for each community. I was surprised that the code has worked the first time and I was pretty happy.

I've played around and tested things and I could get the number of votes and comments on a community. Now all I had to do was to figure out how to update the document and things would work perfectly.

Then everything went wrong!

The voting system was already built as a component so the logical step would be to use this component to not only show the votes but also get the vote from a user. To do this I would have to call the database and update the document to increase the like/dislike vote.

Following the documentation for Node, I was able to write a quick code that should update the database. Since I had to use my API key to initialize the firestore SDK I have decided to use Netlify functions to hide the API Key.

I have tried to use netlify cli and netlify lambda to test my function but for some reason, I was getting a 404 error. I still don't know why this was happening.

Since I couldn't test the netlify function locally I have decided to give it a go and push the changes to try them in production - the site hasn't been that much active and netlify won't publish the latest push if the build fails.

Everything went okay but when I tried to vote nothing has happened. No message got logged and the document remained the same. Since this was the first time I was playing with databases and firestore I just tried a different approach.

That day I stayed awake until 3 am trying to fix the netlify function without any success.

Okay, I wasn't expecting that the code would work first time again. But after following all the suggestions on StackOverflow and the documentation nothing was working.

Testing Locally

The next day I tried to test the function locally using node. Netlify would use Node to run the server-side functions so running the code on my machine should work in production. The code ran without any issues locally, but on Netlify it didn't.

Since I couldn't think what was broken I decided to create a new function and write a lot of console.logs. If I figured out where the function was stopping, I could pin-point where things were broken.

Pushing this new function somehow showed me a new error. I couldn't use firestore SDK with node version less than 8.10 (or something like that). So after a quick search online it seemed I couldn't use the firestore SDK with netlify functions because of the node version.

I wonder why the function didn't complain about this the first time I tried to push the firestore SDK, but maybe I was doing something wrong.

MongoDB Atlas to the rescue

Since I couldn't use google's firestore as my database I had to look for something else. The Makerlog Telegram channel told me that they moved to MongoDB Atlas after struggling with firestore. So, I decided to give it a go since MongoDB is a pretty easy database to query and I had used it in the past while following a tutorial.

After spending so many days fighting with firestore it felt good to just start from scratch and use another technology. The mongodb plugin worked flawlessly and I was able to create a new document on my MongoDB atlas as easy as before with firestore.

I decided to give it a go and write a quick function so I could use node and test to see if I understood the documentation and if I was able to update a document. All worked fine.

Since it worked okay I decided to turn that function into a netlify function so I could use it on the real site. After pushing the changes the database would be called but I couldn't update the document.

Again, I fiddled with stuff with netlify and noticed that even though I was submitting new changes to the function I kept getting the same old logs. It was as if the function wasn't updated.

I've heard tales of how much Netlify caches stuff and I think I have been struck by something like that. After changing the name of my function, I got a different log but the function still wasn't working.

I went back to node and ran my netlify function with a few changes - I added a variable that should update a document on the database. Node complained because I wasn't using await in an async function. This seems pretty obvious but remember that I was struggling with databases for almost a week and my head was a mess.

After fixing that issue I was able to make things work and finally get the vote and update the document on the database.

Afterwards, it was pretty easy to implement the commenting system.

Lesson learned

I've learned a good lesson with this experience. Sometimes cache bites you in your ass and you should plan for it. Also, make sure you check your async code when using .then() and that in some places like netlify functions you won't get any info when you mess your async code.

Finally, thank God node did help and warned me of the async issue otherwise I would still be trying to fight with the function!