how to speed up the code?

Question

in my program i have a method which requires about 4 files to be open each time it is called,as i require to take some data.all this data from the file i have been storing in list for manupalation. I approximatily need to call this method about 10,000 times.which is making my program very slow?

any method for handling this files in a better ways and is storing the whole data in list time consuming what is better alternatives for list?

I can give some code,but my previous question was closed as that only confused everyone as it is a part of big program and need to be explained completely to understand,so i am not giving any code,please suggest ways thinking this as a general question...

thanks in advance

Answer 1

As a general strategy, it's best to keep this data in an in-memory cache if it's static, and relatively small. Then, the 10k calls will read an in-memory cache rather than a file. Much faster.

If you are modifying the data, the alternative might be a database like SQLite, or embedded MS SQL Server (and there are others, too!).

It's not clear what kind of data this is. Is it simple config/properties data? Sometimes you can find libraries to handle the loading/manipulation/storage of this data, and it usually has it's own internal in-memory cache, all you need to do is call one or two functions.

Without more information about the files (how big are they?) and the data (how is it formatted and structured?), it's hard to say more.

Answer 2

Opening, closing, and reading a file 10,000 times is always going to be slow. Can you open the file once, do 10,000 operations on the list, then close the file once?

Answer 3

It might be better to load your data into a database and put some indexes on the database. Then it will be very fast to make simple queries against your data. You don't need a lot of work to set up a database. You can create an SQLite database without requiring a separate process and it doesn't have a complicated installation process.

Answer 4

Call the open to the file from the calling method of the one you want to run. Pass the data as parameters to the method

Answer 5

如果文件是结构化的配置文件，则最好使用ConfigParser库，否则，如果您使用其他结构格式，那么我认为最好将所有这些数据存储为JSON或XML并对数据执行任何必要的操作