Origin Determination
The Find Origins (addon) pipeline stores the candidate origin packages reported by each origin detection tool as origin matches, see OriginMatch. Its last step combines these matches into one origin determination per matched file or directory, see OriginDetermination.
Each determination has a status and the name of the rule that set it. Only an agreement between two tools confirms an origin automatically. Any other case needs a review, with a proposed origin when a rule can select one.
Candidates
The origin matches of a file or directory are grouped by package into candidates:
The matches of the same package version are one candidate, even when the tools report different PURLs, such as
pkg:github/benjaminp/six@1.17.0andpkg:pypi/six@1.17.0, as SCANOSS reports the equivalent PURLs of a component. A version with avprefix, such as thev1.0.3Git tag, is the same as the1.0.3release.The snippet matches covering fewer than 20 lines of the file are not stored as origin matches, such as a common license header found in many unrelated packages. The raw tool results are kept in the project output files.
When nested directories are matched to the same package, the deepest directory is kept as the most precise match.
A file matched only to the package of its closest matched directory has no determination of its own: its matches count for the directory.
Statuses
auto-confirmedSet by the
confirmed-by-two-toolsrule.needs-reviewSet by any other rule. The PURL is the proposed origin, or empty when no candidate is preferred.
confirmedSet by a user who confirms the origin.
no-originSet by a user when the file or directory has no open source origin.
Rules
The rules are evaluated in this order, and the first rule that applies sets the
determination. The candidates that carry the copy of another candidate, as
identified by the containment rule, are not compared by the next rules.
Rule |
Applies when |
Status |
Proposed origin |
|---|---|---|---|
|
Different packages: the copy in all the candidates but one is in a file,
or in a directory just under a vendor directory, named after another
candidate, such as |
|
The remaining candidate. |
|
Versions of the same package: one is more similar than all the others by at least 20 points, or else only one is a release, as opposed to a commit hash or a missing version, or else one release is more similar than the other releases by at least 20 points, or else one release was released before the other releases, as a file often stays the same in several releases. |
|
The preferred version. |
|
Versions of the same package, without a preferred version. |
|
None. |
|
Different packages: one candidate is more similar than all the others by at least 20 points, and all the similarities are known. |
|
The most similar candidate. |
|
Different packages, not resolved by the previous rules. |
|
None. |
|
One candidate, with partial matches only: snippet, approximate, or directory matches. |
|
The candidate. |
|
One candidate, with an exact file or package match, reported by a single tool. |
|
The candidate. |
|
One candidate, with at least one exact file or package match, reported by two tools. |
|
The candidate. |
Similarity and release date
The similarity is the percentage of the file lines matched: the SCANOSS
matchedvalue, 100 for a MatchCode.io exact match, and the line coverage of the MatchCode.io snippet matches. It is unknown for the directory and approximate matches. The tools measure the same copy a few points apart, hence the 20 points margin.The release date is reported by SCANOSS, and by MatchCode.io when PurlDB knows it. It only selects among the versions of the same package, never between different packages: the earliest candidate is often another copy, such as a GitHub repository copying the file, when the original package is not among the candidates.
Re-running the pipeline
The automatic determinations,
auto-confirmedandneeds-review, are replaced.The user determinations,
confirmedandno-origin, are kept. A directory determined by a user covers its content: the origin matches under it are not used.
Origin packages
The files and directories of the auto-confirmed and confirmed
determinations are assigned to the package of their PURL. The files of a
determined directory are assigned with it, unless they have a determination of
their own.
A package detected by the codebase scan is reused, also when its PURL has other qualifiers, such as a Maven
classifier.Otherwise, the package is created from the tools data: download URL, homepage URL, release date, and declared license. It is identified with the
origin_determinationdatasource ID, and created again on each run.
Limitations
A single tool never confirms an origin automatically, as it can miss other candidates: SCANOSS reports one component per file, and MatchCode.io only knows the indexed packages.
A copy taken from a package that vendors it, such as
six.pycopied from urllib3, is attributed to the vendoring package when no tool reports the original package.MatchCode.io rarely reports a release date, so the earliest release rarely selects among the versions of its candidates.
The 20 lines threshold is calibrated on the MIT license header. A longer header, such as the BSD-3-Clause one, still counts as a snippet match.
A small file copied with changes, matched as a snippet on fewer than 20 lines, is ignored.
The
containmentrule needs the path of the copy in each candidate. SCANOSS only reports the file name, and the path of a MatchCode.io match is fetched from PurlDB, when configured. When a PurlDB request fails, the paths are partial and the rule does not apply to the files without a path.