>i "ddlZddlZddlZddlZddlmZddlmZddlZddl m Z m Z m Z ejeZGddeZddeded ed efd Zd ed efd Zded efdZdeed efdZded efdZded efdZded efdZded efdZy)N)unescape) HTMLParser)InvalidBrowserPathInvalidResponse NetworkErrorcTeZdZdZfdZdZdZdZdefdZ de ded e fd Z xZ S) TextExtractorz HTML parser for text extraction. Extracts visible text content from an HTML string, excluding the contents of tags specified in _skip_tags. cPt|g|_d|_hd|_y)NF>stylescripttemplate)super__init___parts_skip _skip_tags)self __class__s Z/var/www/python/tenders/Scrapper/venv/lib/python3.12/site-packages/pydoll/utils/general.pyrzTextExtractor.__init__s#   9c0||jvrd|_yy)z Marks the parser to skip content inside tags specified in _skip_tags. Args: tag (str): The tag name. attrs (list): A list of (attribute, value) pairs. TNrr)rtagattrss rhandle_starttagzTextExtractor.handle_starttags $// !DJ "rc0||jvrd|_yy)zl Marks the parser the end of skip tags. Args: tag (str): The tag name. FNr)rrs r handle_endtagzTextExtractor.handle_endtag(s $// !DJ "rcf|js%|jjt|yy)z Handles text nodes. Adds them to the result unless they are within a skip tag. Args: data (str): The text data. N)rrappendr)rdatas r handle_datazTextExtractor.handle_data2s&zz KK  x~ .rstripc#ZK|jD]}|r|jn|yw)z Yields all collected visible text fragments. Args: strip (bool): Whether to strip leading/trailing whitespace from each fragment. Yields: str: Visible text fragments. N)rr")rr"texts r get_stringszTextExtractor.get_strings<s+KK 2D"'$**,T 1 2s)+ separatorreturncD|j|j|S)a Returns all visible text. Args: separator (str): String inserted between extracted text fragments. strip (bool): Whether to strip whitespace from each fragment. Returns: str: The visible text. )r")joinr%)rr&r"s rget_textzTextExtractor.get_textIs!~~d..U.;<D?CC C CDCD'C55 DDDDpathsc|D]J}tjj|s#tj|tjsH|cSt d|)a Validates potential browser executable paths and returns the first valid one. Checks a list of possible browser binary locations to find an existing, executable browser. This is used by browser-specific subclasses to locate the browser executable when no explicit binary path is provided. Args: paths: List of potential file paths to check for the browser executable. These should be absolute paths appropriate for the current OS. Returns: str: The first valid browser executable path found. Raises: InvalidBrowserPath: If the browser executable is not found at the path. z No valid browser path found in: )ospathisfileaccessX_OKr)rKrNs rvalidate_browser_pathsrRsN$ 77>>$ BIIdBGG$<K ?wG HHrr c,tjdd|tj}tjdd|tj}tjdd|}tjdd|}tjd d |}|S) a Clean JavaScript code by removing comments and string literals. This helps avoid false positives when analyzing script structure. Args: script: JavaScript code to clean. Returns: str: Cleaned script with comments and strings removed. z//.*?$)flagsz /\*.*?\*/z"[^"]*"z""z'[^']*'z''z`[^`]*`z``)resub MULTILINEDOTALL)r cleaneds rclean_script_for_analysisr[soffYF",,?Gff\2wbii@GffZw/GffZw/GffZw/G Nrct|}d}d}ttj||j xs$tj||j S)z Check if a JavaScript script is already wrapped in a function. Args: script: JavaScript code to analyze. Returns: bool: True if script is already a function, False otherwise. z^\s*function\s*\([^)]*\)\s*\{z^\s*\([^)]*\)\s*=>\s*\{)r[r/rVmatchr")r cleaned_scriptfunction_patternarrow_function_patterns ris_script_already_functionras]/v6N77  !>#7#7#9: D 88*N,@,@,B C rct|}t|ryd}tj||sy|j d}d}d}|D]}}tjd|stjd|rd}||j d|j d z z }tj||r |r|dkry|dks|d}y) z Check if a JavaScript script has return statements outside of functions. Args: script: JavaScript code to analyze. Returns: bool: True if script has return outside function, False otherwise. Fz \breturn\b rz \bfunction\bz=>T{})r[rarVsearchsplitcount)r r^return_patternlines brace_count in_functionlines rhas_return_outside_functionrns/v6N".1#N 99^^ 4   &EKK  99_d +ryy/EK tzz#C88  99^T *+"2 ! K " rselectorc|j}|jds|Sd}|j|}|dk(r|S|t|z }|j d}|dk(s||kr|S|||S)a Normalize synthetic XPath selector produced by the builder. Converts selectors of the form //*[@xpath="..."] back into the original XPath string between the quotes. Returns the input unchanged if the pattern is not present or cannot be parsed safely. Args: selector: The selector string that may contain the synthetic XPath format. Returns: str: The normalized original XPath or the input selector if no normalization applies. z //*[@xpath=z //*[@xpath="z"])r" startswithfindlenrfind)rosprefix start_idxend_idxs rnormalize_synthetic_xpathrzsz A << & FvIB VIggdmG"}9, Yw r)rTF)r9loggingrMrVr2r html.parserrr@pydoll.exceptionsrrr getLoggerr+loggerr r0r/r6bytesr<intrJlistrRr[rarnrzrrrs  "OO   8 $E=JE=P===$=SV=" 3# 3% 3HsHsH4I$s)II0cc4st*+++\   r