如果任何作业在 python 中失败，如何退出进程？

Question

I am running jobs in a parallel manner based on sequence number. I am taking the status of every job like - success or Failed. Then after getting the status of every job i am sending mail with status of every job.

But mail is generating after finishing of whole process. But i want if any job will fail the process will stop there and mail will generate.

Can you please help me how to do this?

Code i am running:

        df_mail_final = pd.DataFrame()
        df_mail_final1 = pd.DataFrame()

        '''Getting the status of every job'''
        for m_job in df_main4.master_job.unique():
            list_df = []
            dict_mail = OrderedDict()
            temp_df1 = df_main4[df_main4['master_job'] == m_job].copy()
            temp_df1['duration'] = pd.to_datetime(temp_df1['end_time'].unique()[-1]) - pd.to_datetime(temp_df1['start_time'].unique()[0])
            temp_df1['duration'] = temp_df1['duration'].replace('0 days' ,'')
            status_list = temp_df1.status.unique()
            if(0 in status_list):
                dict_mail['Master Job Name'] = m_job
                idx = temp_df1['status'] == 0
                dict_mail['Execution_Seq'] = temp_df1.loc[idx]["exec_seq"].unique()[0]
                dict_mail['Start_time'] = temp_df1.loc[idx]["start_time"].unique()[0]
                dict_mail['End_time'] = temp_df1.loc[idx]["end_time"].unique()[-1]
                dict_mail['Status'] = 'Failed'
                dict_mail['Duration'] = temp_df1.loc[idx]["duration"].unique()[-1]
                dict_mail['Reason'] = temp_df1.loc[idx]["error_msg"].unique()[0]
                dict_mail['Function_Name'] = temp_df1.loc[idx]["error_func"].unique()[0]
                list_df.append(dict_mail)
                df_mail = pd.DataFrame(list_df)
            if(0 not in status_list):
                print(m_job)
                dict_mail['Master Job Name'] = m_job
                dict_mail['Execution_Seq'] = temp_df1.exec_seq.unique()[0]
                dict_mail['Start_time'] = temp_df1.start_time.unique()[0]
                dict_mail['End_time'] = temp_df1.end_time.unique()[-1]
                dict_mail['Status'] = 'Success'
                dict_mail['Duration'] = temp_df1.duration.unique()[-1]
                dict_mail['Reason'] = ''
                dict_mail['Function_Name'] = ''
                list_df.append(dict_mail)
                df_mail = pd.DataFrame(list_df)
            df_mail_final = pd.concat([df_mail_final,df_mail], axis=0, ignore_index=True)
            #if(df_mail_final['Status'].iloc[-1] == 'Failed'):
                #break

        '''Printing the Final Dataframe with status of all the jobs'''
        print(df_mail_final)
        df_mail_final = df_mail_final[['Master Job Name', 'Execution_Seq', 'Start_time', 'End_time', 'Status', 'Duration', 'Reason', 'Function_Name']]
        exec_end_dt = datetime.datetime.now().strftime("%H:%M:%S")
        #total_duration = pd.to_datetime(exec_end_dt) - pd.to_datetime(exec_start_dt)
        total_duration= pd.to_datetime(df_mail_final['End_time']).max() - pd.to_datetime(df_mail_final['Start_time']).min()
        total_duration = str(total_duration)
        total_duration = total_duration.replace('0 days', '')
        send_mail(df_mail_final, LOG_FILE, total_duration)

Answer 1

Sharing a gist/system design logic of how to implement this job

def my_parallel_job(*args, **kwargs):
    # do your stuff here
    pass

def parallel_job_wrapper(*args, **kwargs):
    try:
       my_parallel_job(*args, **kwargs)
       # if errors following will not run
       return "success"
    except:
      # if errors comes
      return "fail"

def main(*args, **kwargs):
    # call you parallel jobs from here
    p1 = parallel_job_wrapper(*args, **kwargs)
    # preferably you are using something like python's multithreading pool methods

In the above code, the second function is to act as a cushion, in case of any failure of the first function. This ensures that your main does not stop even when any parallel job fails.