topic RE: Remove duplicate rows based on 3 column names in Product Forum

Remove duplicate rows based on 3 column names

Amruthasimplify — Fri, 17 Mar 2023 01:46:00 GMT

Hi All,

Can anyone help me on custom code to remove duplicate rows from a collection based on 3 column names. The input collection has 6 columns in total. i wanted to remove duplicates based on 3 columns.

Thanks in Advance

------------------------------
Amrutha Sivarajan
------------------------------

RE: Remove duplicate rows based on 3 column names

plnarayana777 — Fri, 17 Mar 2023 05:00:00 GMT

Hi Amrutha

Can you try the below code

Dim dt As DataTable = [your DataTable]
Dim columnNames() As String = inputColumnName.Split(","c) 'replace with your input parameter that contains the column names
Dim distinctDT As DataTable = dt.DefaultView.ToTable(True, columnNames)

above code from chat gpt and also refer below thread for better understanding

Delete Repetitive Values in a collection or Excel | Blue Prism Product

Hope this helps

Regards

Lakshmi Narayana

------------------------------
Lakshmi Narayana
------------------------------

RE: Remove duplicate rows based on 3 column names

PvD_SE — Fri, 17 Mar 2023 10:13:00 GMT

Hi Amrutha,

Avoidance of duplicates can be achieved in two ways:

Removing already existing duplicate values
Avoiding getting in duplicate values

While Lakshmi describes how to tackle the first method, you may also consider to have a look at the second method.

If your collection data was derived from an XL, you might consider extracting the data using OLEDB using an SQL DISTINCT statement that, if constructed correctly, will only populate the collection with unique data.

If you data comes from other sources, you might consider checking if a row to be added to the collection perhaps already exists there. And then not adding it if it appears to be a duplicate.

------------------------------
Happy coding!
---------------
Paul
Sweden
------------------------------

RE: Remove duplicate rows based on 3 column names

Mukeshh_k — Fri, 17 Mar 2023 10:23:00 GMT

Hello Amrutha Sivarajan,

You can add an action in one of your extended objects for collection manipulations, make sure you have correct external references for Dll and their Namespaces listed in Initialise stage, please follow below approach:

Add an action in existing collection manipulation extended objects or create a new object if not already.
Write Code as below to act on 3 column based duplicate removal in collections:
System.GC.Collect()
'filteredCollection=RawData.clone()
'filteredCollection=RawData.DefaultView.ToTable(true, columnToCheck)
dim count as integer=rawData.rows.count-1
dim i as integer =0
dim rowIndex as integer
for rowIndex = rawData.rows.count-2 to 0 step rowIndex-1
if rawData.rows(rowIndex).item(columnToCheck1)=rawData.rows(rowIndex+1).item(columnToCheck1) and
rawData.rows(rowIndex).item(columnToCheck2)=rawData.rows(rowIndex+1).item(columnToCheck2) and
rawData.rows(rowIndex).item(columnToCheck3)=rawData.rows(rowIndex+1).item(columnToCheck3)

rawData.rows(rowIndex+1).delete
end if
next
filteredCollection=rawData.copy()
Check required Dll and Namespaces -if you find any errors (Snippet below)

Let me know if you find any difficulties implementing this.

------------------------------
Kindly up vote this as "Best Answer" if it adds value or resolves your query in anyway possible, happy to help.

Regards,

Mukesh Kumar - Senior Automation Developer

NHS England, United Kingdom, GB
------------------------------

RE: Remove duplicate rows based on 3 column names

plnarayana777 — Sat, 18 Mar 2023 04:17:00 GMT

Hi Mukesh

Thanks for your solution

I have tested but their is thing that it is checking for consecutive rows I guess what if their is one row at index like 1 and same is repeated at row index 10

I have implemented and tested it but it is not able to remove duplicate rows which are not in consecutive positions

Can you please check if I am doing any wrong

Regards

------------------------------
Lakshmi Narayana
------------------------------

RE: Remove duplicate rows based on 3 column names

plnarayana777 — Sat, 18 Mar 2023 04:33:00 GMT

Hi Mukesh

Got the answer from your reply in other thread as first need to sort the collection with any one of the column in the three columns

Please ignore the above

Regards

------------------------------
Lakshmi Narayana
------------------------------

RE: Remove duplicate rows based on 3 column names

Mukeshh_k — Sat, 18 Mar 2023 06:43:00 GMT

Glad you were able to sort that out ! There was a similar query but for one column value duplicate - Row removal.

The only thing that changes in the code is the AND condition operator for column/columns to check.