Swift Array firstIndex takes 20 seconds when matching 2 small arrays of structures

Question

I'm using swift / swiftui to assign an structure form one array to an attribute of another structure array. The arrays are fairly small. The figureArray is about 4000 records and the specificsArray is about 200 records. The lookup match firstIndex takes about 20 seconds. Even if I comment out the assignment of specifics to the figureArray , the process takes 20 seconds, which suggests that the firstIndex in a for / forEach is extremely slow.

The memory for the process is about 90M on my iPhone 8 and the CPU hits ~100%

Question is, how can I make it faster? (much faster - ie, less than 2 seconds). To me it seems like this process should take milliseconds for the array sizes.

The specifics objects are unique and never overlap, therefore the setting can be done in parallel. I'm just not sure how.

specificsArray.forEach { specific in
    // look for a figure
    if let indexFigure = figureArray.firstIndex(where: {$0.figureGlobalUniqueId == specific.specificsFirebase.figureGlobalUniqueId}) {
        figureArray[indexFigure].specifics = specific
    }
}

I've also tried the following. The timing is almost identical at about 20 seconds

for indexSpecifics in 0 ..< specificsArray.count {
    // look for a figure
    if let indexFigure = figureArray.firstIndex(where: {$0.figureGlobalUniqueId == specificsArray[indexSpecifics].specificsFirebase.figureGlobalUniqueId}) {
        figureArray[indexFigure].specifics = specificsArray[indexSpecifics]
    }
}

Specific structure

struct Specifics: Hashable, Codable, Identifiable {
    var id: UUID
    var specificsFirebase: SpecificsFirebase
    var isSet = false
}

struct SpecificsFirebase: Hashable, Codable, CustomStringConvertible {
    let seriesUniqueId: String
    let figureGlobalUniqueId: String
    var loose_haveCount: Int = 0
    var loose_sellCount: Int = 0
    var loose_wantCount: Int = 0
    var new_haveCount: Int = 0
    var new_orderCount: Int = 0
    var new_orderText: String = ""
    var new_sellCount: Int = 0
    var new_wantCount: Int = 0
    var notes: String = ""
    var updateDate: String = ""
    
    // print description
    var description: String {
        return ("SpecificsStruct: \(seriesUniqueId), \(figureGlobalUniqueId), \n  LOOSE: Have \(loose_haveCount), sell \(loose_sellCount), want \(loose_wantCount) \n  NEW: Have \(new_haveCount), sell \(new_sellCount), want \(new_wantCount), order \(new_orderCount) \(new_orderText) \n  notes \(notes), update date \(updateDate)")
    }
    
    func saveSpecifics(userID: String) {
        setFirebaseSpecifics(userID: userID)
    }
    
    func setFirebaseSpecifics(userID: String) {
        let firebaseRef: DatabaseReference! = Database.database().reference()
        let specificsPath = SpecificsFirebase.getSpecificsFirebaseRef(userID: userID, seriesUniqueId: SeriesUniqueIdEnum(rawValue: seriesUniqueId)!,
                                                                      figureGlobalUniqueId: figureGlobalUniqueId)
        
        let dateFormatter = DateFormatter()
        dateFormatter.dateFormat = kDateFormatDatabase // firebase 2020-09-13T14:34:47.336
        let updateDate = Date()
        let updateDateString = dateFormatter.string(from: updateDate)

        let firebaseSpecifics = [
            "figureGlobalUniqueId": figureGlobalUniqueId,
            "loose_haveCount": loose_haveCount,
            "loose_sellCount": loose_sellCount,
            "loose_wantCount": loose_wantCount,
            "new_haveCount": new_haveCount,
            "new_orderCount": new_orderCount,
            "new_orderText": new_orderText,
            "new_sellCount": new_sellCount,
            "new_wantCount": new_wantCount,
            "notes": notes,
            "seriesUniqueId": seriesUniqueId,
            "updateDate": updateDateString
        ] as [String: Any]
        
//        #if DEBUG
//        print("Setting firebase specifics for \(firebaseSpecifics)")
//        #endif
        firebaseRef.child(specificsPath).setValue(firebaseSpecifics)
    }
}

Figure structure

struct Figure: Hashable, Codable, Identifiable {
   
    var id = UUID()
//    var id: String { Figure_Unique_ID }
    
    func hash(into hasher: inout Hasher) {
        hasher.combine(figureUniqueId)
    }
    
    let figureGlobalUniqueId: String

    let seriesUniqueId: SeriesUniqueIdEnum
    let figureUniqueId: String
    
    let sortOrder: Int
    let debutYear: Int?
    let phase: String
    let wave: String
    var figureNumber: String?
    let sortGrouping: String
    var uPC: String?
    let figureName: String
    let figurePackageName: String
//    var tags = [String]()
    var scene: String?
    var findTerms: String
    var excludeTerms: String?
    var amazonASIN: String?
    var amazonShortLink: String?
    var walmartSKU: String?
    var targetTCIN: String?
    var targetDPCI: String?
    var entertainmentEarthIN: String?
    var retailDate: Date?
    var retailPrice: Float?
    var addedDate: Date
    
    // calculated or set
    let primaryFrontImageName: String
    let primaryFrontImageNameNoExt: String
    
    // generated retail links
    var searchString: String
    
    // calculated later
    var amazonURL: URL?
    var entertainmentEarthURL: URL?
    var targetURL: URL?
    var walmartURL: URL?
    var eBayURL: URL?

    var specifics: Specifics
    
    init (seriesUniqueId: SeriesUniqueIdEnum,
          figureUniqueId: String,
          sortOrder: Int,
          debutYear: Int?,
          phase: String,
          wave: String,
          figureNumber: String?,
//          sortGrouping: String,
//          tags: String?,
          uPC: String?,
          figureName: String,
          figurePackageName: String,
          scene: String?,
          findTerms: String,
          excludeTerms: String?,
          amazonASIN: String?,
          amazonShortLink: String?,
          walmartSKU: String?,
          targetTCIN: String?,
          targetDPCI: String?,
          entertainmentEarthIN: String?,
          retailDate: Date?,
          retailPrice: Float?,
          addedDate: Date) {
        
        self.seriesUniqueId = seriesUniqueId
        self.figureUniqueId = figureUniqueId
        
        self.figureGlobalUniqueId = "\(seriesUniqueId.rawValue)_\(figureUniqueId)"
        
        self.sortOrder = sortOrder
        self.debutYear = debutYear
        self.phase = phase
        self.wave = wave
        self.figureNumber = figureNumber
        self.sortGrouping = phase // <---------- Uses Phase!
        self.uPC = uPC
        self.figureName = figureName
        self.figurePackageName = figurePackageName
        self.scene = scene
        self.findTerms = findTerms
        self.excludeTerms = excludeTerms
        self.amazonASIN = amazonASIN
        self.amazonShortLink = amazonShortLink
        self.walmartSKU = walmartSKU
        self.targetTCIN = targetTCIN
        self.targetDPCI = targetDPCI
        self.entertainmentEarthIN = entertainmentEarthIN
        self.retailDate = retailDate
        self.retailPrice = retailPrice
        self.addedDate = addedDate
        
        // split out the hash tags
//        if let tags = tags {
//            let words = tags.components(separatedBy: " ")
//             for word in words{
//                 if word.hasPrefix("#"){
////                     let hashtag = word.dropFirst()
//                    self.tags.append(String(word))
//                 }
//             }
//        }
        
        // set the specifics to the default so that the pickers work.  Pickers don't like optionals.
        // DONT SET the isSet here as this is a default record
        self.specifics = Specifics(id: UUID(), specificsFirebase: SpecificsFirebase(seriesUniqueId: seriesUniqueId.rawValue, figureGlobalUniqueId: figureGlobalUniqueId))
        
        // built fields
        self.primaryFrontImageName = "\(seriesUniqueId.rawValue)_\(figureUniqueId)\(kPrimaryFrontImageNameSuffix)\(kSmallSuffix).\(kImageJpgExt)"
        self.primaryFrontImageNameNoExt = "\(seriesUniqueId.rawValue)_\(figureUniqueId)\(kPrimaryFrontImageNameSuffix)\(kSmallSuffix)"
        
        // generated
        self.searchString = "\(seriesUniqueId) \(figureUniqueId), \(phase) \(wave) \(figurePackageName)"
        if let figureNumber = figureNumber {
            self.searchString += " \(figureNumber)"
        }
        if let uPC = uPC {
            self.searchString += " \(uPC)"
        }
        if let amazonASIN = amazonASIN {
            self.searchString += " \(amazonASIN)"
        }
        if let targetTCIN = targetTCIN {
            self.searchString += " \(targetTCIN)"
        }
        if let targetDPCI = targetDPCI {
            self.searchString += " \(targetDPCI)"
        }
        if let entertainmentEarthIN = entertainmentEarthIN {
            self.searchString += " \(entertainmentEarthIN)"
        }
        if let scene = scene {
            self.searchString += " \(scene)"
        }
        if let debutYear = debutYear {
            self.searchString += " \(debutYear)"
        }
    }
    
    enum CodingKeys: String, CodingKey {
        case figureUniqueId = "Figure_Unique_ID"
        case seriesUniqueId = "Series_Unique_ID"
        
        case sortOrder = "Sort_Order"
        case debutYear = "Debut_Year"
        case phase = "Phase"
        case wave = "Wave"
        case figureNumber = "Number"
//        case sortGrouping = "Sort_Grouping"
//        case tags = "Tags"
        case uPC = "UPC"
        case figureName = "Action_Figure"
        case figurePackageName = "Action_Figure_Package_Name"
        case scene = "Scene"
        case findTerms = "Find_Terms"
        case excludeTerms = "Exclude_Terms"
        case amazonASIN = "Amazon_ASIN"
        case amazonShortLink = "Amazon_Short_Link"
        case walmartSKU = "WalmartSKU"
        case targetTCIN = "Target_TCIN"
        case targetDPCI = "Target_DPCI"
        case entertainmentEarthIN = "EEIN"
        case retailDate = "Retail_Date"
        case retailPrice = "Retail_Price"
        case addedDate = "Added_Date"
    }
    
    init(from decoder: Decoder) throws {
        let values = try decoder.container(keyedBy: CodingKeys.self)
        
        let seriesUniqueIdString = try values.decode(String.self, forKey: .seriesUniqueId)
        let figureUniqueId = try values.decode(String.self, forKey: .figureUniqueId)
        let sortOrder = try values.decode(Int.self, forKey: .sortOrder)
        let debutYear = try values.decode(Int.self, forKey: .debutYear)
        let phase = try values.decode(String.self, forKey: .phase)
        let wave = try values.decode(String.self, forKey: .wave)
        let figureNumber = try? values.decode(String.self, forKey: .figureNumber)
//        let sortGrouping = try values.decode(String.self, forKey: .sortGrouping)
//        let tags = try? values.decode(String.self, forKey: .tags)
        let uPC = try? values.decode(String.self, forKey: .uPC)
        let figureName = try values.decode(String.self, forKey: .figureName)
        let figurePackageName = try values.decode(String.self, forKey: .figurePackageName)
        let scene = try? values.decode(String.self, forKey: .scene)
        let findTerms = try values.decode(String.self, forKey: .findTerms)
        let excludeTerms = try? values.decode(String.self, forKey: .excludeTerms)
        let amazonASIN = try? values.decode(String.self, forKey: .amazonASIN)
        let amazonShortLink = try? values.decode(String.self, forKey: .amazonShortLink)
        let walmartSKU = try? values.decode(String.self, forKey: .walmartSKU)
        let targetTCIN = try? values.decode(String.self, forKey: .targetTCIN)
        let targetDPCI = try? values.decode(String.self, forKey: .targetDPCI)
        let entertainmentEarthIN = try? values.decode(String.self, forKey: .entertainmentEarthIN)
        let retailDateString = try? values.decode(String.self, forKey: .retailDate)
        let retailPrice = try? values.decode(Float.self, forKey: .retailPrice)
        let addedDateString = try? values.decode(String.self, forKey: .addedDate)
        
        // calculated
        let seriesUniqueId = SeriesUniqueIdEnum(rawValue: seriesUniqueIdString)!
        
        // date logic
        let dateFormatter = DateFormatter()
        dateFormatter.dateFormat = "MM/dd/yyyy" //Your date format
        
        var retailDate: Date? = nil
        if let retailDateString = retailDateString {
            if let retailDateValid = dateFormatter.date(from: retailDateString) {
                retailDate = retailDateValid
            }
        }
        
        var addedDate = defaultAddedDate
        
        if let addedDateString = addedDateString {
            if let addedDateValid = dateFormatter.date(from: addedDateString) {
                addedDate = addedDateValid
            }
        }
        
        self.init(seriesUniqueId: seriesUniqueId,
                  figureUniqueId: figureUniqueId,
                  sortOrder: sortOrder,
                  debutYear: debutYear,
                  phase: phase,
                  wave: wave,
                  figureNumber: figureNumber,
//                  sortGrouping: phase,
//                  tags: tags,
                  uPC: uPC,
                  figureName: figureName,
                  figurePackageName: figurePackageName,
                  scene: scene,
                  findTerms: findTerms,
                  excludeTerms: excludeTerms,
                  amazonASIN: amazonASIN,
                  amazonShortLink: amazonShortLink,
                  walmartSKU: walmartSKU,
                  targetTCIN: targetTCIN,
                  targetDPCI: targetDPCI,
                  entertainmentEarthIN: entertainmentEarthIN,
                  retailDate: retailDate,
                  retailPrice: retailPrice,
                  addedDate: addedDate)
    }
    
    func encode( to encoder: Encoder) throws {
        var container = encoder.container(keyedBy: CodingKeys.self)
        try container.encode(self.seriesUniqueId, forKey: .seriesUniqueId)
        try container.encode(self.figureUniqueId, forKey: .figureUniqueId)
        try container.encode(self.sortOrder, forKey: .sortOrder)
        try container.encode(self.debutYear, forKey: .debutYear)
        try container.encode(self.phase, forKey: .phase)
        try container.encode(self.wave, forKey: .wave)
        try container.encode(self.figureNumber, forKey: .figureNumber)
        //        try container.encode(self.sortGrouping, forKey: .sortGrouping)
        //        try container.encode(self.tags, forKey: .tags)
        try container.encode(self.uPC, forKey: .uPC)
        try container.encode(self.figureName, forKey: .figureName)
        try container.encode(self.figurePackageName, forKey: .figurePackageName)
        try container.encode(self.scene, forKey: .scene)
        try container.encode(self.findTerms, forKey: .findTerms)
        try container.encode(self.excludeTerms, forKey: .excludeTerms)
        try container.encode(self.amazonASIN, forKey: .amazonASIN)
        try container.encode(self.amazonShortLink, forKey: .amazonShortLink)
        try container.encode(self.walmartSKU, forKey: .walmartSKU)
        try container.encode(self.targetTCIN, forKey: .targetTCIN)
        try container.encode(self.targetDPCI, forKey: .targetDPCI)
        try container.encode(self.entertainmentEarthIN, forKey: .entertainmentEarthIN)
        try container.encode(self.retailDate, forKey: .retailDate)
        try container.encode(self.retailPrice, forKey: .retailPrice)
        try container.encode(self.addedDate, forKey: .addedDate)
    }
}

Answer 1

The problem here is that your algorithm is running in quadratic time. For each element in one array you are linearly searching through the other array. Worst case, that means every element of the second array is compared to every element of the first array. (x * y comparisons!)

This should help:

func example(specificsArray: [Specifics], figureArray: inout [Figure]) {
    let specificsDict = Dictionary(uniqueKeysWithValues: specificsArray
        .map { ($0.specificsFirebase.figureGlobalUniqueId, $0) })
    for (index, figure) in figureArray.enumerated() {
        if let specific = specificsDict[figure.figureGlobalUniqueId] {
            figureArray[index].specifics = specific
        }
    }
}

The above will drop the big-O time to linear (assuming a good hash.) The code is no longer comparing each specific with every figure. Instead, it is doing a hash calculation and looking up the specific in constant time (ideally at least.) This is at the cost of running through the specifics once to create the dictionary which is also linear.

Another benefit is that neither Specifics nor Figure need to be Hashable.

Try it out and see if you get a performance improvement.

Answer 2

The problem is that you're likely making a full copy of figureArray every time you mutate it. Swift value types are "copy on write," which means they may be copied anytime they're mutated. Often this can be avoided, though. This may work dramatically better if you turn on optimizations (ie build for Release), but in Debug mode it's possible that it can't avoid the copying.

One way to avoid that is to turn this around, and iterate over figureArray rather than over specificsArray . I would expect this is easier to optimize. It also avoids searching the big array so many times. This touches every element of the big array exactly one time rather than half of the elements for every element of specificsArray:

for index in figureArray.indices {
    let id = figureArray[index].figureGlobalUniqueId
    if let specific = specific.first(where: { id == specific.specificsFirebase.figureGlobalUniqueId }) {
        figureArray[index].specifics = specific
    }
}

That should hopefully avoid any copying, but if it doesn't, you can ensure that there there is exactly one copy rather than many by turning it into a map:

figureArray = figureArray.map { figure in
    guard let specific = specificsArray.first(where: { specific in 
        specific.specificsFirebase.figureGlobalUniqueId == figure.figureGlobalUniqueId }) 
    else { return figure }  // Return the original value if nothing has changed

    // Otherwise update it    
    var newFigure = figure
    figure.specifics = figure
    return figure
}

When you made this a class, you made copying the array much cheaper. The struct is very large, so copying it is expensive. When you copy an array of classes, you only have to add an extra retain count on each and copy a pointer. That can be much faster when the struct is massive. (But it would be better to avoid all the copying instead if possible.)