r/learnpython•
TheSlutyCam

help to define MVC to cron jobs with SQLAlchemy

I have 3 different cron jobs that are pieces of the same program. It's far to be completed, but I start to get errors, using raw sqlite functions, because of the mixed connections from each cron to the same DB, that I decided to review the mess.

So I tough in change all the structure, and put the all thing run smoothing with an MVC model and switch to SQLAlchemy to accomplish that. But it's hard for me to implement that, because I really don't know where to begin.

Where should I put the DB code? And the different functional or trivial functions that bind each other? And if exists cron jobs what are the best practices and advice's? And how to avoid the connections errors when they hit the same DB from different crons ?!?

I'm pretty sure that I'm not the only one lost on this.

To understand better the SQLAlchemy thing, all I can find is Django or Flask tutorials to build web Apps. And for cron jobs, seems too much just to see/ read the results. Not suitable or fitting for crons, that's what I think now. Or is it?

But to get that conclusion, I have learned some Flask, I have build my App that allow to CRUD the data of the crones, even by binding with already existing DB using the SQLAlchemy. The thing is that in never interacts with my previous code, except with the DB.

So it's not enough for what I intend to understand, learn and apply in my personal project.

Short story, what I'm looking for, is a guide/ tutorial and your help, to rearrange all the pieces/ code, in the right place, to control things better, allow the project to grow when needed. The idea is to be a cron for ever, all I need to view are the errors and the results.

I can share here the list of the *.py files and all they're functions, but let's start in abstract mode.

what i have:

  • file with all sqlite3 instructions and functions, to create, read, clean, insert in each and at each table from 4 DB's to serve all the cron's 2900 lines of code (could be more if I repeated this functions in each cron, right ?

  • ** file with basic functions, to serve all the crons, again to avoid repetition in each cron they are all together. No sqlite here, just settings, dates function, scrape_settings and so on**

  • cron_One_Time_A_Day - configure_log_and_save_traceback, create_tables, clean_old_records, scrape, save_scrape, redefine_time_for_next_scrape, scrape_the_catalog_to_the_next_crons, save_the_catalog, couple_extra_functions_to_avoid_nested_ifs, main_to_bind_all_this_and_start_the_cron

  • cron_Each_5_Mins - configure_log_and_save_traceback, create_tables, clean_old_records, get_scrape_list_and_time_to_scrape_from_DB, get_the_list_of_sub_scrapes_from_previous_list, scrape_that_and_save_in_DB, couple_extra_functions_to_avoid_nested_ifs_and_big_mess, pre_main_to_bind_if_its_time_to_scrape_or_exit, main_to_bind_all_the_mess

  • cron_2_Times_Each_Minute - configure_log_and_save_traceback, create_tables, clean_old_records, scrape, save_the_scrape_in_DB, check_for_changes_with_previous_records, get_details_from_each_change, couple_extra_functions_to_help_with_details, and_some_more_functions_to_avoid_big_mess_and_nested_ifs, some_math_and_save_in_DB

  • to_implement - cron_Each_Minute - configure_log_and_save_traceback, create_tables, clean_old_records, get_scrape_from_DB, sensitize_details, extra_couple_functions_to_help_with_that, sensitize_more_details, store_all_that_details, another_function_get_all_that_new_compacted_details, scrape_recent_details_because_of_that, save_the_scrape, think_about_that_scrape, more_extra_functions_to_help_think, execute_instructions_based_on_that, main_to_bind_all_that

Thanks for any time, patience and guiding for this topic ;)

Posted Aug 29, 2016, 10:52 PM UTC

10
Loading comments…