Improve the performances of the project details view to load faster by deferring the the charts rendering. This is especially noticeable on projects with a large amount of codebase resources and discovered packages. https://github.com/nexB/scancode.io/issues/193
Add support for filtering by “Other” values when filtering from the charts in the Project details view. https://github.com/nexB/scancode.io/issues/526
CodebaseResource.for_packagesnow returns a list of
DiscoveredPackage.package_uidis not present. This is done to reflect the how scancode-toolkit’s JSON output returns
package_uid``s in the ``for_packagesfield for Resources.
Add the model DiscoveredDependency. This represents Package dependencies discovered in a Project. The
scan_packagespipelines have been updated to create DiscoveredDepdendency objects. The Project API has been updated with new fields:
dependency_count- The number of DiscoveredDependencies associated with the project.
discovered_dependencies_summary- A mapping that contains following fields:
total- The number of DiscoveredDependencies associated with the project.
is_runtime- The number of runtime dependencies.
is_optional- The number of optional dependencies.
is_resolved- The number of resolved dependencies.
These values are also available on the Project view. https://github.com/nexB/scancode.io/issues/447
dependenciesfield has been removed from the DiscoveredPackage model.
Create directory CodebaseResources in the rootfs pipeline. https://github.com/nexB/scancode.io/issues/515
Add ProjectErrors when the DiscoveredPackage could not be fetched using the provided package_uid during the assemble_package step instead of failing the whole pipeline. https://github.com/nexB/scancode.io/issues/525
Escape paths before using them in regular expressions in
Disable multiprocessing and threading by default on macOS (“spawn” start method). https://github.com/nexB/scancode.io/issues/522
WARNING: Drop support for Python 3.6 and 3.7. Add support for Python 3.10. Upgrade Django to version 4.1 series.
Upgrade ScanCode-toolkit to version 31.0.x. See https://github.com/nexB/scancode-toolkit/blob/develop/CHANGELOG.rst for an overview of the changes in the v31 compared to v30.
Ensure the worker service waits for migrations completion before starting. To solve this issue we install the wait-for-it script available in Debian by @vishnubob and as suggested in the Docker documentation. In the docker-compose.yml, we let the worker wait for the web processing to be complete when gunicorn exposes port 8000 and web container is available. Reference: https://docs.docker.com/compose/startup-order/ Reference: https://github.com/vishnubob/wait-for-it Reference: https://tracker.debian.org/pkg/wait-for-it https://github.com/nexB/scancode.io/issues/387
Add a “create-user” management command to create new user with its API key. https://github.com/nexB/scancode.io/issues/458
Add a “tag” field on the CodebaseResource model. The layer details are stored in this field in the “docker” pipeline. https://github.com/nexB/scancode.io/issues/443
Add support for multiple inputs in the LoadInventory pipeline. https://github.com/nexB/scancode.io/issues/451
Add new SCANCODEIO_REDIS_PASSWORD environment variable and setting to optionally set Redis instance password.
Ensure a project cannot be deleted through the API while a pipeline is running. https://github.com/nexB/scancode.io/issues/402
Display “License clarity” and “Scan summary” values as new panel in the project details view. The summary is generated during the scan_package pipeline. https://github.com/nexB/scancode.io/issues/411
Enhance Project list view page:
20 projects are now displayed per page
Creation date displayed under the project name
Add ability to sort by date and name
Add ability to filter by pipeline type
Add ability to filter by run status
Correctly extract symlinks in docker images. We now use the latest container-inspector to fix symlinks extraction in docker image tarballs. In particular broken symlinks are not treated as an error anymore and symlinks are extracted correctly. https://github.com/nexB/scancode.io/issues/471 https://github.com/nexB/scancode.io/issues/407
Add a Package details view including all model fields and resources. Display only 5 resources per package in the list view. https://github.com/nexB/scancode.io/issues/164 https://github.com/nexB/scancode.io/issues/464
Add the ability to filter by empty and none values providing the “EMPTY” magic value to any filters. https://github.com/nexB/scancode.io/issues/296
CodebaseResource.name now contains both the bare file name with extension, as opposed to just the bare file name without extension. Using a name stripped from its extension was something that was not used in other AboutCode project or tools. https://github.com/nexB/scancode.io/issues/467
Export current results as XLSX for resource, packages, and errors list views. https://github.com/nexB/scancode.io/issues/48
Add support for .tgz extension for input files in Docker pipeline https://github.com/nexB/scancode.io/issues/499
Add support for resource missing file content in details view. Refine the annotation using the new className instead of type. https://github.com/nexB/scancode.io/issues/495
Change the worksheet names in XLSX output, using the “PACKAGES”, “RESOURCES”, “DEPENDENCIES”, and “ERRORS” names. https://github.com/nexB/scancode.io/issues/511
Update application Package scanning step to reflect the updates in scancode-toolkit package scanning.
Package data detected from a file are now stored on the CodebaseResource.package_data field.
A second processing step is now done after scanning for Package data, where Package Resources are determined and DiscoveredPackages and DiscoveredDependencies are created.
Add authentication for the Web UI views and REST API endpoint. The autentication is disabled by default and can be enabled using the SCANCODEIO_REQUIRE_AUTHENTICATION settings. When enabled, users have to authenticate through a login form in the Web UI, or using their API Key in the REST API. The API Key can be viewed in the Web UI “Profile settings” view ince logged-in. Users can be created using the Django “createsuperuser” management command. https://github.com/nexB/scancode.io/issues/359
Include project errors in XLSX results output. https://github.com/nexB/scancode.io/issues/364
Add input_sources used to fetch inputs to JSON results output. https://github.com/nexB/scancode.io/issues/351
Refactor the update_or_create_package pipe to support the ProjectError system and fix a database transaction error. https://github.com/nexB/scancode.io/issues/381
Add webhook subscription available when creating project from REST API. https://github.com/nexB/scancode.io/issues/98
Add the project “reset” feature in the UI, CLI, and REST API. https://github.com/nexB/scancode.io/issues/375
Add a new GitHub action that build the docker-compose images and run the test suite. This ensure that the app is properly working and tested when running with Docker. https://github.com/nexB/scancode.io/issues/367
Add –no-install-recommends in the Dockerfile apt-get install and add the linux-image-amd64 package. This packages makes available the kernels required by extractcode and libguestfs for proper VM images extraction. https://github.com/nexB/scancode.io/issues/367
Add a new list-project CLI command to list projects. https://github.com/nexB/scancode.io/issues/365
Remove the –no-install-recommends in the Dockerfile apt-get install to include required dependencies for proper VM extraction. https://github.com/nexB/scancode.io/issues/367
Synchronize QUEUED and RUNNING pipeline runs with their related worker jobs during worker maintenance tasks scheduled every 10 minutes. If a container was taken down while a pipeline was running, or if pipeline process was killed unexpectedly, that pipeline run status will be updated to a FAILED state during the next maintenance tasks. QUEUED pipeline will be restored in the queue as the worker redis cache backend data is now persistent and reloaded on starting the image. Note that internaly, a running job emits a “heartbeat” every 60 seconds to let all the workers know that it is properly running. After 90 seconds without any heartbeats, a worker will determine that the job is not active anymore and that job will be moved to the failed registry during the worker maintenance tasks. The pipeline run will be updated as well to reflect this failure in the Web UI, the REST API, and the command line interface. https://github.com/nexB/scancode.io/issues/130
Enable redis data persistence using the “Append Only File” with the default policy of fsync every second in the docker-compose. https://github.com/nexB/scancode.io/issues/130
Add a new tutorial chapter about license policies and compliance alerts. https://github.com/nexB/scancode.io/issues/337
Include layers in docker image data. https://github.com/nexB/scancode.io/issues/175
Fix a server error on resource details view when the compliance alert is “missing”. https://github.com/nexB/scancode.io/issues/344
Migrate the ScanCodebase pipeline from scancode.run_scancode subprocess to scancode.scan_for_application_packages and scancode.scan_for_files. https://github.com/nexB/scancode.io/issues/340
Fix a build failure related to dependency conflict. https://github.com/nexB/scancode.io/issues/342
Upgrade ScanCode-toolkit to version 30.1.0
Replace the task queue system, from Celery to RQ. https://github.com/nexB/scancode.io/issues/176
Add ability to delete “not started” and “queued” pipeline tasks. https://github.com/nexB/scancode.io/issues/176
Add ability to stop “running” pipeline tasks. https://github.com/nexB/scancode.io/issues/176
Refactor the “execute” management command and add support for –async mode. https://github.com/nexB/scancode.io/issues/130
Include codebase resource data in the details of package creation project errors. https://github.com/nexB/scancode.io/issues/208
Add a SCANCODEIO_REST_API_PAGE_SIZE setting to control the number of objects returned per page in the REST API. https://github.com/nexB/scancode.io/issues/328
Provide an “add input” action on the Project endpoint of the REST API. https://github.com/nexB/scancode.io/issues/318
Add ability to “archive” projects, from the Web UI, API and command line interface. Data cleanup of the project’s input, codebase, and output directories is available during the archive operation. Archived projects cannot be modified anymore and are hidden by default from the project list. A project cannot be archived if one of its related run is queued or already running. https://github.com/nexB/scancode.io/issues/312
Remove the run_extractcode pipe in favor of extractcode API. https://github.com/nexB/scancode.io/issues/312
The scancode.run_scancode pipe now uses an optimal number of available CPUs for multiprocessing by default. The exact number of parallel processes available to ScanCode.io can be defined using the SCANCODEIO_PROCESSES setting. https://github.com/nexB/scancode.io/issues/302
Renamed the SCANCODE_DEFAULT_OPTIONS setting to SCANCODE_TOOLKIT_CLI_OPTIONS. https://github.com/nexB/scancode.io/issues/302
Log the outputs of run_scancode as progress indication. https://github.com/nexB/scancode.io/issues/300
Upgrade ScanCode-toolkit to version 21.7.30
Add new documentation chapters and tutorials on the usage of the Web User Interface. https://github.com/nexB/scancode.io/issues/241
Add ability to register custom pipelines through a new SCANCODEIO_PIPELINES_DIRS setting. https://github.com/nexB/scancode.io/issues/237
Add a pipeline scan_package.ScanPackage to scan a single package archive with ScanCode-toolkit. https://github.com/nexB/scancode.io/issues/25
Detected Package dependencies are not created as Package instance anymore but stored on the Package model itself in a new dependencies field. https://github.com/nexB/scancode.io/issues/228
Add the extra_data field on the DiscoveredPackage model. https://github.com/nexB/scancode.io/issues/191
Improve XLSX creation. We now check that the content is correctly added before calling XlsxWriter and report and error if the truncated can be truncated. https://github.com/nexB/scancode.io/issues/206
Add support for VMWare Photon-based Docker images and rootfs. This is an RPM-based Linux distribution
Add support for VM image formats extraction such as VMDK, VDI and QCOW. See https://github.com/nexB/extractcode#archive-format-kind-file_system for the full list of supported extensions. The new extraction feature requires the installation of libguestfs-tools, see https://github.com/nexB/extractcode#adding-support-for-vm-images-extraction for installation details. https://github.com/nexB/scancode.io/issues/132
Add the ability to disable multiprocessing and threading entirely through the SCANCODEIO_PROCESSES setting. Use 0 to disable multiprocessing and use -1 to also disable threading. https://github.com/nexB/scancode.io/issues/185
Missing project workspace are restored on reports (xlsx, json) creation. This allow to download reports even if the project workspace (input, codebase) was deleted. https://github.com/nexB/scancode.io/issues/154
Add ability to search on all list views. https://github.com/nexB/scancode.io/issues/184
Add the is_binary, is_text, and is_archive fields to the CodebaseResource model. https://github.com/nexB/scancode.io/issues/75
Adds a new way to fetch docker images using skopeo provided as a plugin using docker:// reference URL-like pointers to a docker image. The syntax is docker://<docker image> where <docker image> is the string that would be used in a “docker pull <docker image>” command. Also rename scanpipe.pipes.fetch.download() to fetch_http() https://github.com/nexB/scancode.io/issues/174
Pipeline status modals are now loaded asynchronously and available from the project list view.
Fix an issue accessing codebase resource content using the scan_codebase and load_inventory pipelines. https://github.com/nexB/scancode.io/issues/147
The installation local timezone can be configured using the TIME_ZONE setting. The current timezone in now included in the dates representation in the web UI. https://github.com/nexB/scancode.io/issues/142
Fix pipeline failure issue related to the assignment of un-saved (not valid) packages. https://github.com/nexB/scancode.io/issues/162
Add a new QUEUED status to differentiate a pipeline that is in the queue for execution from a pipeline execution not requested yet. https://github.com/nexB/scancode.io/issues/130
Refactor the multiprocessing code for file and package scanning. All database related operation are now executed in the main process as forking the existing database connection in sub-processes is a source of issues. Add progress logging for scan_for_files and scan_for_application_packages pipes. https://github.com/nexB/scancode.io/issues/145
Links from the charts to the resources list are now also filtered by in_package/not_in_package if enabled on the project details view. https://github.com/nexB/scancode.io/issues/124
Add ability to filter on codebase resource detected values such as licenses, copyrights, holders, authors, emails, and urls. https://github.com/nexB/scancode.io/issues/153
Filtered list views from a click on chart sections can now be opened in a new tab using ctrl/meta + click. https://github.com/nexB/scancode.io/issues/125
Add links to codebase resource and to discovered packages in list views.
Add support for Docker and VM images using RPMs such as Fedora, CentOS, RHEL, and openSUSE linux distributions. https://github.com/nexB/scancode.io/issues/6
Add a compliance alert system based on license policies provided through a policies.yml file. The compliance alerts are computed from the license_expression and stored on the codebase resource. When the policy feature is enabled, the compliance alert values are displayed in the UI and returned in all the downloadable results. The enable and setup the policy feature, refer to https://scancodeio.readthedocs.io/en/latest/scancodeio-settings.html#scancode-io-settings https://github.com/nexB/scancode.io/issues/90
Add a new codebase resource detail view including the file content. Detected value can be displayed as annotation in the file source. https://github.com/nexB/scancode.io/issues/102
Download URLs can be provided as inputs on the project form. Each URL is fetched and added to the project input directory. https://github.com/nexB/scancode.io/issues/100
Run celery worker with the “threads” pool implementation. Implement parallelization with ProcessPoolExecutor for file and package scans. Add a SCANCODEIO_PROCESSES settings to control the multiprocessing CPUs count. https://github.com/nexB/scancode.io/issues/70
Optimize “tag” type pipes using the update() API in place of save() on the QuerySet iteration. https://github.com/nexB/scancode.io/issues/70
Use the extractcode API for the Docker pipeline. This change helps with performance and results consistency between pipelines. https://github.com/nexB/scancode.io/issues/70
Implement cache to prevent scanning multiple times a duplicated codebase resource. https://github.com/nexB/scancode.io/issues/70
Create the virtualenv using the virtualenv.pyz app in place of the bundled “venv”. https://github.com/nexB/scancode.io/issues/104
Consistent ordering for the pipelines, now sorted alphabetically.
Display project extra data in the project details view. https://github.com/nexB/scancode.io/issues/88
Add a @profile decorator for profiling pipeline step execution. https://github.com/nexB/scancode.io/issues/73
Support inputs as tarballs in root_filesystem pipelines. The input archives are now extracted with extractcode to the codebase/ directory. https://github.com/nexB/scancode.io/issues/96
Improve support for unknown distros in docker and root_filesystem pipelines. The pipeline logs the distro errors on the project instead of failing. https://github.com/nexB/scancode.io/issues/97
Implement Pipeline registration through distribution entry points. Pipeline can now be installed as part of external libraries. With this change pipelines are no longer referenced by the Python script path, but by their registered name. This is a breaking command line API change. https://github.com/nexB/scancode.io/issues/91
Add a “Run Pipeline” button in the Pipeline modal of the Project details view. Pipelines can now be added from the Project details view. https://github.com/nexB/scancode.io/issues/84
Upgrade scancode-toolkit to version 21.2.9
Allow to start the pipeline run immediately on addition in the add_pipeline action of the Project API endpoint. https://github.com/nexB/scancode.io/issues/92
Rename the pipes.outputs module to pipes.output for consistency.
Remove the dependency on Metaflow. WARNING: The new Pipelines syntax is not backward compatible with v1.0.x https://github.com/nexB/scancode.io/issues/82
Add user interface to manage Projects from a web browser All the command-line features are available https://github.com/nexB/scancode.io/issues/24
Log messages from Pipeline execution on a new Run instance log field https://github.com/nexB/scancode.io/issues/66
Add support for scancode pipes and Project name with whitespaces
Add a profile() method on the Run model for profiling pipeline execution https://github.com/nexB/scancode.io/issues/73
Add a management command to delete a Project and its related work directories https://github.com/nexB/scancode.io/issues/65
Add CSV and XLSX support for the output management command https://github.com/nexB/scancode.io/issues/46
Add a to_xlsx output pipe returning XLSX compatible content https://github.com/nexB/scancode.io/issues/46
Add a “status” management command to display Project status information https://github.com/nexB/scancode.io/issues/66
Fix the env_file location to run commands from outside the root dir https://github.com/nexB/scancode.io/issues/64
Add utilities to save project error in the database during Pipeline execution https://github.com/nexB/scancode.io/issues/64
Install psycopg2-binary instead of psycopg2 on non-Linux platforms https://github.com/nexB/scancode.io/issues/64
Add minimal license list and text views https://github.com/nexB/scancode.io/issues/32
Add admin actions to export selected objects to CSV and JSON The output content, such as included fields, can be configured for CSV format https://github.com/nexB/scancode.io/issues/48 https://github.com/nexB/scancode.io/issues/49
Add –list option to the graph management command. Multiple graphs can now be generated at once.
Add ProjectCodebase to help walk and navigate Project CodebaseResource loaded from the Database Add also a get_tree function compatible with scanpipe.CodebaseResource and commoncode.Resource https://github.com/nexB/scancode.io/issues/52
Add support for running ScanCode.io as a Docker image https://github.com/nexB/scancode.io/issues/9
Add support for Python 3.7, 3.8, and 3.9 https://github.com/nexB/scancode.io/issues/54
Add a to_json output pipe returning ScanCode compatible content https://github.com/nexB/scancode.io/issues/45
Improve Admin UI for efficient review: display, navigation, filters, and ability to view file content https://github.com/nexB/scancode.io/issues/36
Add Pipelines and Pipes documentation using Sphinx autodoc Fix for https://github.com/nexB/scancode.io/issues/38
Add new ScanCodebase pipeline for codebase scan Fix for https://github.com/nexB/scancode.io/issues/37
Upgrade Django, Metaflow, and ScanCode-toolkit to latest versions
Add ability to resume a failed pipeline from the run management command Fix for https://github.com/nexB/scancode.io/issues/22
Use project name as argument to run a pipeline Fix for https://github.com/nexB/scancode.io/issues/18
Add support for “failed” task_output in Run.get_run_id method Fix for https://github.com/nexB/scancode.io/issues/17
Add documentation and tutorial For https://github.com/nexB/scancode.io/issues/8
Add a create-project, add-input, add-pipeline, run, output management commands to expose ScanPipe features through the command line Fix for https://github.com/nexB/scancode.io/issues/13
Always return the Pipeline subclass/implementation from the module inspection Fix for https://github.com/nexB/scancode.io/issues/11
Do not fail when collecting system packages in Ubuntu docker images for layers that do not install packages by updating to a newer version of ScanCode Toolkit Fix for https://github.com/nexB/scancode.io/issues/1