v1.1 - PageSearch update

PageSearch mode

For complex OSINT, it's not enough to find only vectors for their further use. Sensitive or other specific information that interests investigator is also important for more accurate approaches. We've added PageSearch feature, which allows you to find more e-mail addresses, API keys, exposed passwords, cookies, hidden forms of data and other web page elements, documents, config files, databases files and specified words by user in PDF files. Although PageSearch will continue to develop, you can use it to get some interesting results in v1.1

XLSX and HTML report support

We decided to add a little variety when you choose the type of report which DPULSE generates. XLSX is an extremely common and no less convenient format than PDF for creating reports. In our XLSX report file you can see several sheets, each of which contains some information

Also, HTML reports support was added. We had some problems with encodings in xhtml2pdf library and PDF reports, so HTML reports are more like an alternative way to get correct output in final report file. In further releases PDF report most likely will be removed, but if solution will appear - PDF reports remain

No more wkhtmltopdf installation requirement

Installing various third-party programs while working with another software is extremely inconvenient. We decided to rework the process of PDF reports creation using a different set of libraries that do not require installation of wkhtmltopdf. This will allow you to start using DPULSE much faster right after downloading it

Scan journal

In situations when errors appear, user wants to know what caused it. With this idea, scan log was introduced. It collects all successful scanning phase and various errors in a separate journal.log file. You can find it in DPULSE root folder after conducting at least one scan

Rolling releases branch

When v1.1 will be released, we will create new branch in GitHub repository. It will contain rolling releases which are being developed in live mode.

v1.0.5

ADDED

1. Added some checks on empty inputs in scanning section of main menu
2. Added e-mails return from PageSearch to use them in reports
3. Added new function (PageSearch Sitemap Inspection) to inspect sitemap links
4. Added .json, SQL and different config variants files extraction in PageSearch
5. Added post-scan summary in PageSearch and PageSearch Sitemap Inspection
6. Added "documents with keywords" counter in PageSearch
7. Added custom KeyboardInterrupt handler (#60)
8. Added exception handler for situations when organization name cannot be extracted (#61, thanks to jdenoy)
9. Added pages separators for PDF report
10. Added 3 additional web elements to search (PageSearch)
11. Added counter of inspected links (PageSearch Sitemap nspection)
12. Added txt log of gathered data (PageSearch Sitemap Inspection)
13. Added sitemap_links.txt file generation when choosing XLSX report (#64), which allows you to use PageSearch Sitemap Inspection mode
14. Added checks on invalid menu items
15. Added logs_processing.py to handle scan logging (#58)
16. Added logging for crawl_processor.py functions (#58)
17. Added logging for networking_processor.py functions (#58)
18. Added appropriate log file name field in XLSX and PDF reports
19. Added API keys and cookies gathering in PageSearch

REMOVED

1. Removed "=>" symbol when writing parsed sitemap links in file
2. Removed "Sitemap links" sheet in XLSX report (#64)

CHANGED

1. Fixed bug when function waits for e-mails return but PS was not conducted
2. Fixed bug when PageSearch ends program because subdomains weren't found
3. Fixed CLI typo in DB menu
4. Edited DPULSE wiki link
5. Updated "Help" menu content
6. Renamed "Scanning process" to "Basic scanning process"
7. Minor CLI visual improvements

v1.0.4

ADDED

1. Added PageSearch module (currently in BETA testing stage)
2. Created reporting_modules folder with all modules which handle report creation
3. Added sys link to reporting_modules folder in dpulse.py
4. Added PageSearch basic support
5. Added PageSearch basic functionality with modules in pagesearch folder
6. Added crawl_processor.py text fillers when DPULSE cant find certain domain element
7. Added networking_processor.py text fillers when DPULSE cant find element
8. Added exception handlers from issues #44, #45
9. Added text filler if domain doesn't have contact e-mails (#46)
10. Added text fillers to subdomain's PDF report part (#47)
11. Added PyMuPDF requirement
12. Added PDF keywords search function for PageSearch
13. Added possibility to decline keywords search but continue docs extraction (#53)
14. Added "separate by comma" remark in keywords input
15. Added check on corrupted PDF files downloaded by PageSearch
16. Added exception handler to subdomains gathering module (#55)
17. Added check for PageSearch if subdomains were not found
18. Added info about PageSearch in general information (part of #43)
19. Added support of transfering PageSearch status message to html template (part of #43)
20. Filled empty gaps when DPULSE can't find some elements of domain (#34)
21. Added more exception handlers (#44, #45)

CHANGED

1. Moved pdf_report_creation.py from datagather_modules folder
2. Moved xlsx_report_creation.py from datagather_modules folder
3. Fixed dependabot alerts in requirements.txt, pyproject.toml and poetry.lock
4. Fixed xlsx report filling issue (#49)
5. Fixed contact e-mails representation in PDF report (part of #42)
6. Fixed MX records not displayed in PDF report (#30)
7. Changed color style of main menu
8. Changed date and time format to international
9. Normalized subdomains emails output for PDF report (part of #42)
10. Normalized output of InternetDB results (part of #42)
11. Fixed bug when 1 letter takes 1 string in PDF report (#40)
12. Code clean-ups, more cosmetical improvements (#42), minor interface changes and so on

v1.0.3

ADDED

1. Added new data_assembler.py module which calls information gathering functions and assembles their results into one list, which then is used by report creation modules
2. Added xhtml2pdf requirement
3. Created .bat installer for Windows systems
4. Created .sh installer for Linux systems
5. Created run function in dpulse.py to support Poetry entry point

REMOVED

1. Removed DPULSE dependence on wkhtmltopdf third-party program, so you don't need to install it in DPULSE root anymore
2. Removed code part which checks wkhtmltopdf.exe presence in DPULSE root
3. Removed pdfkit requirement and imports from modules

CHANGED

1. Some cosmetical changes in report_template.html
2. Changed both xlsx and pdf report creation modules to correctly interact with data_assembler.py module
3. Changed some other modules to correctly interact with data_assembler.py module
4. Changed urllib3 required version to 2.2.2
5. Fixed wrong entry point in pyproject.toml file

v1.0.2-minor

NETWORKING PROCESSOR

1. Fixed error when get_sitemap_xml function couldn't get sitemap.xml by adding some more checks
2. Modified sitemap gathering function to interact with xlsx_report_creation.py module

XLSX REPORT CREATION PROCESSOR

1. Created xlsx_report_creation.py module which will handle XLSX report creation and its writing to report storage database

DPULSE MAIN

1. Added xlsx report file type selection

REPORT STORAGE DATABASE

1. Redacted SQL database creation script by adding column with report file extension
2. Output sequence when choosing print in CLI
3. Added some exception handlers (recreate report with id which is not in db)

DORKING PROCESSOR

1. Added transfer_results_to_xlsx function to

COMMAND LINE INTERFACE

1. Extended output of database content with report file extension (XLSX or PDF)
2. Slightly changed output of DB content while choosing manage database menu

FILES PROCESSING MODULE

1. Renamed "get_pdf_blob" function to "get_blob"

REQUIREMENTS

1. Updated Jinja2 requirement to version 3.1.4 in order to path CVE-2024-34064 vulnerability
2. Added openpyxl 3.1.2 requirement

v1.0.1

So, release is here. We've also released on PyPI (https://pypi.org/project/dpulse/), you can download it from here also.

Many bugs were fixed, some things were redacted, but there's a lot of work to do further. Hope you'll support us and DPULSE with stars and enjoy the time with our program

v0.9-release-candidate-3

ADDED

1. Added programm check if there is some data in report storage database. If there's nothing in DB - program won't allow you to use report recreation function and print informational message that DB is empty
2. Added exception handler for DB report recreation in case you'll try to recreate report ID which folder already exists in DPULSE root

DELETED

1. Deleted read_config function in pdf_report_creation.py

CHANGED

1. Changed program structure by adding service folder with new db_processing.py and files_processing.py modules
2. Some CLI changes and improvements
3. Moved DB processing functions to service/db_processing.py
4. Moved some parts of code from pdf_report_creation.py to db_processing.py and files_processing.py in service folder

v0.9-release-candidate-2

ADDED

1. Added colorama requirement in README
2. Added function to write additional 4 report text files into report_storage.db
3. Added some new string in DB CLI menu to work with new DB features (indicators whether the new files are in DB)

DELETED

1. Some points were deleted from Help menu
2. Deleted some points from GitHub Wiki

CHANGED

1. Changed figlet font to slant
2. whois was replaced with python-whois in requirements.txt
3. report_creation.py module was renamed in pdf_report_creation.py, all imports were also renamed and since that this module will handle only report creation with PDF file included (preparation to xlsx report processing module)
4. Database report extraction now creates folder with report_extracted_ID#N name and puts PDF report and 4 text files in it

v0.9-release-candidate-1

ADDED

1. Added social meadias links counter in PDF report (general information section)
2. Added social medias duplicates deleting code
3. Added function to add Google Dorks in config.txt
4. Added import sys to every module
5. Added import error handlers in every module
6. Added additional return in solid_google_dorking function in error handling block
7. Added error handler in extract_links_from_sitemap in networking_processor

DELETED

1. Removed fake-useragent and useragent libraries from requirements.txt and setup.py
2. Removed error code wiki part

CHANGED

1. Changed style of welcome text in welcome_menu function in cli_init.py module in order to fix situations when zones without text where white-colored
2. Some minor CLI improvements
3. Moved spinner class to dpulse.py (fixed error which led to program close after second scan attempt)
4. Moved import time, import threading and import itertools from cli_init.py to dpulse.py
5. Fixed wrong formattion of social medias table in PDF report
6. Some visual improvements
7. Error codes were replaced with "Reason" message, which means you will see error which causes the handler

v0.8b_hotfix_1 (HEARTBEAT)

Added lxml requirement to setup.py and requirements.txt