ItSucks

ItSucks is a cool Java web spider, or web crawler, that lets you download files and even pick up where you left off if something goes wrong. It's super flexible, too! You can customize it with regular expressions and download templates to fit your needs.

Easy to Use Interface

This application has a swing GUI for those who love visuals and a console interface for the command-line fans out there. Plus, all the backend features are available in a separate library, so you can use them for other projects easily!

Key Features of ItSucks

Here are some key features of ItSucks:

• Supports multiple protocols; right now, it works with http/https.


• You can run several crawl jobs at the same time.


• The number of working threads for each crawl job is adjustable (though this isn’t available in the GUI yet).


• Search depth can be set to whatever you need.


• You can prioritize individual URLs based on your preferences.


• A customizable processing chain helps manage downloaded data efficiently.


• You have the option to save and load one or multiple jobs (serialize/deserialize).


• Built using Java 1.5 features like generics.


• You can configure allowed hostnames with regular expressions.


• The types of files to download are also customizable via regular expressions.


• A base URL helps focus on URLs that start with specific prefixes.

Advanced Filtering Options

• An advanced filter allows multiple regular expressions with actions defined for when they match or don’t match URLs.


Possible actions include:


- Follow the URL (Accept)



- Don’t follow the URL (Reject)



- Change URL priority


​​​​​​
• A filter can also add more parameters into your job!

​​​​​​
• You can track every event fired by the framework and filter events by category/type!

Download Management Made Simple

• Keep an eye on multiple downloads at once.


• Add or remove download jobs as needed!


• An integrated regex tester helps you debug your regular expressions directly!


• Load and save downloads as templates!


• Start your download templates from the console after creating them in the GUI!

Requirements

Your setup needs:

• Java JRE version 1.5 or newer


• At least 256 MB RAM (the more, the better since Java loves memory!)