one solution for big data is: using databases. So when you want to work with AWS, use dynamoDB. If you don't want to handle dynamoDB, you can use graphql with aws amplify. Pretty useful if you ask me. Like that your data is there in less than 500ms

Actually, we have to select a protein from dropdown.
And for each protein, there is specific .csv file.
That file contains the data of that protein.

Since proteins are made up of amino acids, which are made up of Nitrogens, carbons, etc..

So csv files contains all those atoms, there respective postion, respective branches, and probabilites of mutation amino acid, and many other stuff.

So its quite large file.

And we are not fetching same attributes everytime from those csv files, thats why we always need to load the complete csv

in future there might be more dropdown options, hence more csv files.
Is storing all these in a database better option than storing them in S3 / cloudfront?

yes

Since you are having to fetch the entire file, parse it, and only return some attributes from it, you are essentially querying the file. It'd be much faster to query a database

umm this makes sense

which services are cheaper though:
S3+cloudfront
OR
dynamoDB

OR
graphQL + amplify

Also,
these csv files have so much data, is there any way of transferring the content from csv to database? or do I have to do it manually?

yes sql databases will support csv import for sure, since csvs are already in row format

and which one to use? dynamoDB or something else?

dynamoDB is a key value store, a noSQL database

according to you...what would be the best option, considering the rates of the services

sql database for sure

i'm not familiar with amplify, and S3+cloudfront isn't sql

databases are waaaay more effective. So in almost every case you will be cheaper with a database

I am already using sqlite as database to store user credential details and stuff.
so should i add those csv files data in it somewhere?

You can use MySQL or Postgres

yes sqlite should work for you

will I face any performance issues if i do it on sqlite? considering that in future, there will be more csv files (a typical csv file is about 25MB)

how many MB are you fetching in a single request?

it depends on what option user is selecting

there are csv files ranging from 5-50MB

and i am basically querying those csv files, where my required data is, i find it and then show it on frontend

actually, i dont want to give full access to my csv files...

otherwise i would have gave them to download the csv and they could compute things on there on

but that data in csv was generated by our ML models, and we dont want to make it public

right the file is 5-50MB. You can compare that to a SQL table. But you're not returning the entire file's worth of data in a single request are you?

So you'd query the database table (file), and only retgurn some of it

any sql database can do this with ease

so any sql db will save me time as compared to s3

any idea how fast would that be?

yes, it will be extremely fast compared to your current method

@Dwarf Crocodile if you index properly it will be significantly faster

for csv files..there are like 10k rows

probably dont even need to index specific fields, it'll just be much faster

yes, db is designed exactly for that

any tutorials on how to do it?

flatfile csv isnt made for querying

you have no sql/mysql knowledge?

not enough

i just know how to make schema and store stuff

thats all you need

and a query

just make it basic, it will be fast enough im sure

idk anyting about indexing

well even if you have an incremental id as your primary key index

and thats it

it'll still be way faster

and what is redis?

for caching n stuff?

thats a usecase for redis yeah , think of redis as an in memory solution

so your sessions and stuff like that you can use redis

your data sounds relational, so i would definitely suggest sql db solution

no need for caching?

no not unless you see a need

and if you do, you dont need redis just to cache

you can cache it on your server

for example nextJS caches fetch endpoints anyway iirc

i am using react n flask

n sqlite db

you should convert that csv to sql

and store it in your db

query it and see how much faster iti s

hmm

ok

will try

not the same thing but i have an app that generates PDFs on the fly

previously i was generating them each time, because i thought they're small

but now i store them in db

generating and serving took like a second, db lookup is instant

like a few ms

ohhh

thats reallly good

yeah trust me when u have large data

database is the best

okk

and i just need to make a separate table for each csv file

yeah and think of the relationships between each table

doesnt have to be anything special

umm

ok

thnx 🫡

np

gl noob let me know how it goes

i will ask you guys doubts, if I get any while working on this

👍

what u do btw?

wym

for work?

im a frontend dev

but i like backend also lol

good

hey, 1 more doubt.

2-3 of my csv files have about 20k rows

ain't that wayy too much?

no wonder its slow af lol

you have to remember

hmm

this is exactly what db is designed for

20k is not a lot by db standards

ohh

but for csv it definitely is

so querying a 100 row db table take similar time as querying a 10k row db table?

i meant, querying 10k will still take under 1 sec?

depends if you're querying an index, in which case yes, but in most cases you're not doing that

but it wont be slow

yeah it should be very very fast

cauz now, it takes about 7-10sec

hmm

cool

everytime you're querying it?

thats insane

sql will be much faster

Got it

should def. use it then

you have someone here handling 500 million on a pretty old pc

waow

like most scalable leaderboard systum uses redis...to show ranking in realtime

idk for what reason they use redis

redis is useful because its in memory

but yeah...they also handle pretty large data

i dont get it... "in memory"

a leaderboard constantly changes and needs computation, your data is .csv files so it wont even change

in memory = RAM

if something is a file its not being stored on your RAM its on your hard drive

you get it?

yes yes

ohh

like when you are on google chrome, the tabs you're on, they're "in memory"

hmm....chrome tabs take up a lot of my RAM 😦

hmmm

haha lol same