使用 python 处理大量数据的有效方法是什么？

Question

I have a large set of data consisting of sensor measurements over time stored into multiple Excel and CSV files. A batch of files was created each day. I need to be able to exctract the data from a particular day, a particular file, a particular time and a particular sensor.

The data is displayed like this in each file:

[DD/MM/YYYY hh:mm:ss, "Sensor1", "Sensor2", ..., "SensorX"]

I already have a code to extract said data but only for a specified day, the code is using the panda module to create a dataframe from an excel file and then use it to perform calculations.

What I want to do is to find a way to rapidly extract part of this data. eg : I want to have the measures from sensor 1 and 34 between 01/01/2020 00:00:30 and 05/01/2020 06:00:00.

I was thinking about creating a 3D dataframe with the following dimensions: (Sensor name, time of the day, day), but I don't really know how to do it, is there a way to add a day index to navigate through the data frame? Something like this:

(Day 1 [DD/MM/YYYY hh:mm:ss, "Sensor1", "Sensor2", ..., "SensorX"], Day 2 [DD/MM/YYYY hh:mm:ss, "Sensor1", "Sensor2", ..., "SensorX"], ... Day X [DD/MM/YYYY hh:mm:ss, "Sensor1", "Sensor2", ..., "SensorX"])

But I am afraid that even when filtering some of the data (I don't need the data of every sensors at once), the code will be very slow.

Also each file doesn't contain the same amount of data eg day1.xlsx is 110x8400 and day2.xlsx is 110x5419, etc... Is this an issue, do I need to be particularly careful?

Finally, do you have any recommendations about dealing with large amounts of data? For context, I will have to compare this experimental data with physical models results. Is there a better way than using pandas dataframes?

I know that many answers are somewhere on the inte.net but I have a lot to do aside and even if I'm used to programming, I have never used many libraries in python, so I already spend a lot of time reading documentation, etc...

That's why I'm asking here for insights, so thank you in advance, I hope it all makes sense for you, English is not my first language so please indulge my mistakes.

Have a nice day !

Edit: I need to use python since I can't install third party programs and I will later use models using the python environment.

Answer 1

I need to be able to exctract the data from a particular day, a particular file, a particular time and a particular sensor.

To me this has database written all over the place. There are several time series specific DBs, such as InfluxDB, TimescaleDB or IoTDB. I would recommend you to look at IoTDB as it seems to be targeting your usecase, ie sensor measurements, as they they in their front page:

IoTDB is a specialized database management system for time series data generated by a.network of IoT devices with low computational power.

Alternatively you can take a regular SQL database, for example SQLite if you think you will not have huge amounts of data and if you will not need other people being able to access it from some remote server. If you want something more powerful PostgreSQL is always a good choice too.

With SQL database you will be able to query your sensor measurements in many different ways, combining different filters and grouping/aggregations of data in case if you need that too.